feat(bench): v2 republish — post-SQLR-23 pinned-host run + headline update (SQLR-25)#114
Merged
Merged
Conversation
…pdate (SQLR-25) Reruns the full 12-workload suite (SQLRite + SQLite + DuckDB on Group B) on the canonical Apple M1 Pro / macOS 23.5.0 host after SQLR-23 (parser plan cache + bound `?` params + `Value::Vector`) bumped every workload from `v1 → v2` and SQLR-28 took W10 → `v3`. The v1 numbers in `docs/benchmarks.md` were dominated by per-iter SQL parsing on the bench driver's SQLRite path; v2 measures the post-binding reality. Headline: **W10 HNSW: 126.81 ms → 2.40 ms (~53× faster)**, with brute-force-vs-HNSW gap now ~50× at 10k corpus (was ~1.1× in v1, masked by parser tax + L2-only probe). Other parser-tax-dominated paths tightened: W1 read-by-PK ~2.5×, W6 secondary-index ~2.6×, W3 bulk insert ~1.7×, W11 BM25 top-10 ~2.2× faster than v1. Compute-bound paths (W4 single-row insert, W5 mixed OLTP, W7 SUM, W8 GROUP BY) moved a few percent — parser cost was a small slice of total time there. Changes: - `benchmarks/results/2026-05-08-apple-ac84d560.json` — canonical v2 envelope, 35 samples (Group A SQLRite+SQLite + Group B SQLRite+SQLite+DuckDB + Group C SQLRite-only). - `docs/benchmarks.md` — Headline numbers now reflect v2; v1 retained in a "Historical (v1, retired)" section so the methodology shift is visible. "Parser tax" sidebar updated to point at the new run. - `benchmarks/README.md` — adds W10.v3 result table (the v1 retired table stays alongside) + status banner bumped to 2026-05-08. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
?params +Value::Vector) and SQLR-28 (HNSW cosine/dot probe) have landed. Every workload'sWorkloadId.versionwas bumpedv1 → v2in lockstep (W10 → v3); v1 numbers are retained as historical so the methodology shift is captured.Files
benchmarks/results/2026-05-08-apple-ac84d560.json— canonical v2 envelope, 35 samples (Group A SQLRite+SQLite + Group B SQLRite+SQLite+DuckDB + Group C SQLRite-only).docs/benchmarks.md— Headline numbers now reflect v2; v1 retained in a "Historical (v1, retired)" subsection. Parser-tax sidebar updated to point at the new run.benchmarks/README.md— adds W10.v3 result block alongside the retired v1 table; status banner bumped to 2026-05-08.Methodology note
target/criterion/retained staleW*.v1directories from prior runs; the aggregator scrapes everything in that tree, so the first attempt produced a polluted envelope with v1 numbers copied through. Killed, removed the staleW*.v1*dirs, rebuilt clean. Worth hardening inscripts/run.shlater (out of scope here — filed in the project journal).Test plan
cargo fmt --all -- --checkcargo build --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --exclude sqlrite-benchmarks --all-targetscargo test --workspace --exclude sqlrite-desktop --exclude sqlrite-python --exclude sqlrite-nodejs --exclude sqlrite-benchmarks(all suites + doctests pass)make bench-duckdbproduced the canonical envelope on the pinned host (Apple M1 Pro/macos/aarch64, run started2026-05-08T15:41:43Z, 35 samples)benchmarks/scripts/compare.py v1 v2Q8-flags the cross-version pairs (W*.v1vsW*.v2/v3) in the "Only on one side" section as designed🤖 Generated with Claude Code